Created to manage the process to build, train, tune, deploy, and manage machine models.
MLflow, an open source machine learning (ML) platform created by Databricks, will join the Linux Foundation.
Since its introduction at Spark + AI Summit two years ago, MLflow has about 200 contributors and is downloaded about “two million times per month”.
Databricks created MLflow in response to the “complicated process” of ML model development. Traditionally, the process to build, train, tune, deploy, and manage machine models was extremely difficult for data scientists and developers.
Databricks co-founder and CTO Matei Zaharia created MLflow to also track versions of data sets, model parameters, and algorithms, which creates an exponentially larger set of variables to track and manage.
In addition, ML is very iterative and relies on close collaboration between data teams and application teams, he said.
MLflow keeps this process from becoming overwhelming by providing a platform to manage the end-to-end ML development lifecycle from data preparation to production deployment, including experiment tracking, packaging code into reproducible runs, and model sharing and collaboration.
“Machine learning is transforming all major industries and driving billions of decisions in retail, finance, and health care,” Zaharia said. “Our move to contribute MLflow to the Linux Foundation is an invitation to the machine learning community to incorporate the best practices for ML engineering into a standard platform that is open, collaborative, and end-to-end.”
The Linux Foundation provides a vendor neutral home with an open governance model to broaden adoption and contributions to the MLflow project even further, said Michael Dolan, VP of Strategic Programs at the Linux Foundation.
“The steady increase in community engagement shows the commitment data teams have to building the machine learning platform of the future. The rate of adoption demonstrates the need for an open source approach to standardising the machine learning lifecycle,” he said.
